Skip a replicated publish the receiving node has already logged, so a restart or reconnect doesn't deliver it twice by ldt1996 · Pull Request #4339 · HarperFast/harper

ldt1996 · 2026-11-12T22:03:33Z

⊙ Problem

On 5.4, after a restart, worker restart or replication reconnect, a node delivers replicated MQTT messages to its live subscribers a second time: its resume re-sends the log below the last certified floor (harper-pro#1092), and a re-sent publish was logged and delivered again.

💡 Solution

Stacked on #4384, which keeps a record's version from moving back.

A replicated publish is skipped when its origin's log here already holds it (same key, table, record key and version): nothing is logged or delivered (check, lookup).
Only a message at or below its topic's record version is looked up, so an in-order publish reads no log; a missing record is looked up only on a table that can drop records (guard).
A message the node never applied is still applied once; local publishes and LMDB are unchanged. Release note; DESIGN.md.

⚖️ Alternatives

A rocksdb-js committed-identity lookup (planning review: option set too narrow) — would also close the concurrent-apply race, but needs a native release; that race only duplicates (Two concurrent applies of one replicated transaction can both log it, because re-delivery checks read only up to the log's committed-read watermark #4340).
harper-pro's leading-duplicate fast-skip — armed only for the subscription's own node; core's apply owns duplicate detection.
Dedupe at delivery — lost on a node restart, and the copies would still be logged and relayed.
Resume at the applied position — loses chore: bump lmdb to 3.5.5 #1092's re-send of a late commit below it.

🤔 Where a re-sent message is still applied again

A duplicate, never a loss: two applies of one origin transaction racing (#4340); an entry purged by audit retention (#4348); markers configured off; a topic record an earlier version moved back, or a deleted topic whose tombstone was cleaned up.

🤔 Decision: one narrow loss case is accepted

5.3 wrote no abort markers. If a 5.3 apply logged a message and failed for good just before the upgrade, and that transaction is re-sent after another write moved its topic's record past it, the message is skipped. Trusting no pre-restart file would close this, but would re-apply every superseded message after each restart until those files are purged.

✅ Verification

replicatedPublishRedelivery.test.js: re-delivery cases fail on main and pass; new cases check that an in-order publish to a held topic and the first publish to a new topic read no log (test). relayedHeadTieDuplicate.test.js expects a re-delivered publish to read its entry once.
test:unit:resources 8,194 pass on RocksDB, test:unit:lmdb 6,098; failing in both runs only rocksOpenConfig, LMDB's startTime replay handoff and Replay a long run of patches to a record in linear time #4337's bounded-history case, which fail on main too, and originClosedFloor, which passes in isolation.
Two harper-pro nodes on one host, QoS 0 at 20 messages/s: base gave 111 subscriber duplicates after the publisher restarted and logged 103–111 messages twice after a worker or node restart; the fix gave 0. Deliver replicated messages to subscribers once after a restart or replication reconnect harper-pro#1362 covers the same restarts.

Performance: no cost on in-order publishes (min µs/op over 7×20k replicated publishes, two rounds): held topic 17.5–18.0 vs main 17.9–18.0; new topic 16.8–16.9 vs 16.8–16.9. The unstacked check cost 20.3–20.7 on the same held-topic run.

PR QA: skipped @ 6a06693 — dispatch is not available in this session

_{🤖 Generated by Claude Opus 5.5 (Claude Code); posted via @ldt1996.}

Release target: v5.4 — fixes a regression in the 5.4 betas (harper-pro#1092's floor-only resume); 5.3 has no floor-only resume

Framing-Verdict: option-set-too-narrow (c11e475ee1be)

Related PRs: #4384 overlaps (this PR's base: a record's version never moves back); #3137 overlaps (publish delivery in resources/Table.ts); 2 others independent
Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex,gemini-carbon; late=gemini-carbon:adjudicated; adjudicated=domain; blocked=cursor-composer(not-installed); declined=cursor-grok,muse; rounds=4; full=2 @ 6a06693}

_{Review-Attention: deep ~30m (critical: Table.ts; decisions: do-less-alternative, requirement-shift, uncertified-history-trust, perf-cost; raised: degraded review) @ 6a06693}

kriszyp

Thanks for running this down. Probing the origin's log at the incoming key is the right way to tell a duplicate from a late message, and the restart and reconnect numbers make the case well.

But the premise the per-publish probe rests on: "a late one lowers the record's version". On RocksDB, _writePublish restamps the record unconditionally: recordUpdater stores newVersion as the record's version and log reference (RecordEncoder.ts, "with rocksdb, we simplify to just storing the singular version/timestamp"). A replicated message also runs no precedesExistingVersion check. So a late publish moves the record's version and log reference backward. I think that backward move is the defect, and the duplicates are one symptom of it:

The record then points at an older log entry than the latest message for that topic, until the next publish.
A later replicated put compares its version with the lowered one. Example: a put at 200 is applied, a late publish at 150 lowers the record to 150, then a late put at 170 wins on this node (precedesExistingVersion, txnTime <= existingEntry.version), while the origin keeps the 200 value. We're checking this with a repro now, so treat it as unconfirmed.
If a publish only ever moves the record's version forward (keep the maximum, while still logging and delivering the late message), dedup can check the version first:

a message newer than the record's version is new, so no probe;
one at or below it is late or a duplicate, so probe the origin's log exactly as this PR does.
That takes the 2.3 µs off every in-order publish and makes #4341 unnecessary. It would also cover the "a later-arriving message lowered the record" case listed under "Where a re-sent message is still applied again". The open question is whether walking a topic's message history through its previous-version links still finds a late message once the record's reference never points at it.

I think it would be difficult to justify this significant of performance regression, and so I think we should address the root cause instead. Let me know if you want any help with this. I'm also drafting a PR with a test to reproduce the lower-level issue.

Drafted with a little bit of help from Opus 5.5.

kriszyp · 2026-11-13T00:20:08Z

+							options?.nodeId,
+							options?.version ?? txnTime
+						) &&
+						(auditHistoryAbortMarked(auditStore, options?.nodeId) ||


[major] Retained old files also allow duplicate delivery of messages committed into newer files

kriszyp · 2026-11-13T14:28:46Z

Here is a reproduction demonstrating this is a inbound replication handling issue, not a subscription issue: #4360

…king its version first A checkpoint-grade resume re-sends the origin's log below its certified floor, so a node re-applied publishes it had already logged and delivered them to its subscribers again. A replicated publish is now skipped when the origin's log on this node already holds that message (same key, table, record id and version), read once per applying transaction. Stacked on #4384, which keeps a record's version from moving back, so only a message at or below the record's version is looked up and an in-order publish reads no log. A retry never skips a publish whose first attempt staged its entry, and in a log file from before this process only a record vouches for an entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e the check applies An exact-start miss below a log's first entry scans to its end, so a re-delivery there (retention purged it, or a late-relayed entry appended below the first) is applied again rather than probed, as _writeUpdate's dedup does. The release note no longer claims every in-order message is free: the first to a topic with no record is still looked up. DESIGN.md records when a replicated publish is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… the first-entry shortcut A log's first appended entry is not a lower bound on its keys, so skipping the lookup below it delivered a retained late message twice; it is removed. The exact-start probe repeated the seek the range read makes, so the range read alone answers whether the origin transaction is logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…cords On a table that keeps its records, a topic's record outlives its messages, so a missing record means a new topic and no logged message: the first publish to it reads no log. A table that evicts or expires records, or keeps no audit log, still checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… logged Submodule core -> 6a0669337 (HarperFast/harper#4339, stacked on #4384). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kriszyp

_{🤖 Reviewed with Codex}

kriszyp · 2026-11-14T15:48:42Z

+						abortMarkersEnabled() &&
+						!(retry && write.appendedAuditEntry === true) &&
+						(existingEntry === undefined
+							? (evictsRecords() || !audit) && auditHistoryAbortMarked(auditStore, options?.nodeId)


[major] Newly committed messages can still be delivered twice after restart

The missing-topic trigger from the prior finding remains. On an expiring or evicting audited table, retain an origin's pre-restart file, roll into a new file, commit message A after restart onto a populated topic, then evict that record without purging A's log entry. Re-send A: this branch refuses the lookup because auditHistoryAbortMarked() remains false while the older file survives (resources/auditStore.ts:1913-1922). A is logged and delivered again even though its own batch was written with abort markers and committed in the newer file.

Preserving record versions fixes the late-publish trigger, but cannot supply evidence after eviction removes the record. Scope trust to the matched frame's file/sequence, potentially through a rocksdb-js committed-identity primitive. The discriminating check is that restart/rollover/apply/evict/re-send sequence, expecting one delivery of A. The simulated old-history case at unitTests/resources/replicatedPublishRedelivery.test.js:283-302 retains the topic record, so it does not exercise this branch. This remaining outcome is supported by code tracing; I could not run a database repro in this read-only checkout.

kriszyp · 2026-11-14T15:48:42Z

@@ -11351,10 +11356,30 @@ export function makeTable(options): TableResourceClass {
 					(this.constructor as any).source?.publish && !(context as any)?.source


[major] Newly committed messages can still be delivered twice after restart

_{(resources/Table.ts:11158 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

… logged Submodule core -> 6a0669337 (HarperFast/harper#4339, stacked on #4384). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ldt1996 requested a review from kriszyp November 12, 2026 22:03

ldt1996 added this to the v5.4 milestone Nov 12, 2026

ldt1996 marked this pull request as ready for review November 12, 2026 23:54

ldt1996 mentioned this pull request Nov 13, 2026

A replicated MQTT message resent after audit retention removed its log entry is delivered again #4348

Open

ldt1996 force-pushed the fix/replicated-publish-redelivery branch from a299f1b to e8e5f81 Compare November 13, 2026 12:51

kriszyp requested changes Nov 13, 2026

View reviewed changes

kriszyp mentioned this pull request Nov 13, 2026

Add known-failure tests showing a late publish moves its record's version backward #4360

Closed

ldt1996 changed the title ~~Deliver replicated messages to subscribers once after a restart or replication reconnect~~ Skip a replicated publish the receiving node has already logged, so a restart or reconnect doesn't deliver it twice Nov 13, 2026

ldt1996 mentioned this pull request Nov 13, 2026

Keep a record's version when a message older than it is published #4384

Merged

Lavinia and others added 4 commits November 14, 2026 15:57

ldt1996 force-pushed the fix/replicated-publish-redelivery branch from e8e5f81 to 6a06693 Compare November 14, 2026 15:41

ldt1996 changed the base branch from main to fix/late-publish-keeps-record-version November 14, 2026 15:41

kriszyp approved these changes Nov 15, 2026

View reviewed changes

kriszyp merged commit 57c1e38 into fix/late-publish-keeps-record-version Nov 15, 2026
98 of 101 checks passed

kriszyp deleted the fix/replicated-publish-redelivery branch November 15, 2026 02:00

Navigation Menu

Repository navigation

Skip a replicated publish the receiving node has already logged, so a restart or reconnect doesn't deliver it twice - #4339

Skip a replicated publish the receiving node has already logged, so a restart or reconnect doesn't deliver it twice#4339
kriszyp merged 4 commits into
fix/late-publish-keeps-record-versionfrom
fix/replicated-publish-redelivery

ldt1996 commented Nov 12, 2026 •

edited

Loading

Uh oh!

kriszyp left a comment

Uh oh!

kriszyp Nov 13, 2026

Uh oh!

kriszyp commented Nov 13, 2026

Uh oh!

kriszyp left a comment

Uh oh!

kriszyp Nov 14, 2026

Uh oh!

kriszyp Nov 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -11351,10 +11356,30 @@ export function makeTable(options): TableResourceClass {
		(this.constructor as any).source?.publish && !(context as any)?.source

Conversation

ldt1996 commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⊙ Problem

💡 Solution

⚖️ Alternatives

🤔 Where a re-sent message is still applied again

🤔 Decision: one narrow loss case is accepted

✅ Verification

Uh oh!

kriszyp left a comment

Choose a reason for hiding this comment

Uh oh!

kriszyp Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

kriszyp commented Nov 13, 2026

Uh oh!

kriszyp left a comment

Choose a reason for hiding this comment

Uh oh!

kriszyp Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

kriszyp Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ldt1996 commented Nov 12, 2026 •

edited

Loading